Hyperspectral Imaging Spectrophotometer and System

ABSTRACT

A hyperspectral imaging spectrophotometer and system, with calibration, data collection, and image processing methods designed to match human visual perception and color matching of complex colored objects.

The present application relates to a hyperspectral imagingspectrophotometer and system, with calibration, data collection, andimage processing methods designed to match human visual perception andcolor matching of complex colored objects.

BACKGROUND

Retail paint stores typically offer paint matching services, whereinpaints are custom formulated to match a color of a sample inspirationitem. The inspiration item may be a paint chip, a fabric, or an object.

In one example, a customer goes to a hardware store and would like tohave a custom can of paint mixed that matches a physical color patchfrom a fan deck or other physical sample. In the workflow, thecustomer's color patch is first measured by an instrument such as aspectrophotometer or colorimeter, that is capable of outputtingcalibrated colorimetric data such as a spectral reflectance curve, ortri-stimulus values in a color space such as CIE L*a*b*. Oncecolorimetric data is obtained for the color patch, the values can beused as inputs to algorithms that use paint databases and computeeither, a) the database paint color that most closely matches that ofthe patch, or b) a recipe of colorants and quantities from analgorithmic formulation in which to blend with a quantity of paint basematerial in order to make a can of paint that matches the customer'ssample.

However, retail paint stores have difficulty matching 10-25% ofinspiration samples. The mismatches result in increased labor, delay indelivery time, inaccurate paint matching, wastage of paint and reducedcustomer satisfaction. Existing spectrophotometers, though fairlyaccurate on planar uniform-color paint-like samples, are unable tosatisfactorily measure many textured samples, samples with defects,samples with multiple colored regions that need to be discriminated andevaluated separately, and samples with other colorimetric spatialcomplexity. They also are difficult or impossible to use on multi-colorsamples, such as patterned fabric or wall paper. This is becauseexisting spectrophotometers are spot measurement devices that computethe average color over a contiguous region of a sample. All of thespatial color information is lost because the average reduces themeasurement to a single number. Further, since spatial information isaveraged into a single measurement, prior art instrumentation cannotremove defects, make statistical analysis of sub-regions, separate ameasurement into dominant colors, remove gloss or shadows that are notpart of the sample's core color, or apply spatial transforms relevant tomodelling human visual perception.

SUMMARY

A hyperspectral imaging system for measuring one or more colors of asample surface, may have a sample measurement area, an illuminationassembly, a pick-up optics assembly, a main processor, and an image dataprocessor. The illumination assembly includes at least one illuminatordirected toward the sample measurement area. The pick-up optics assemblyincludes a plurality of selectable spectral filters having differentnominal peak wavelengths and an imaging camera, where a field of view ofthe imaging camera includes the sample measurement area. The mainprocessor is coupled to the illumination assembly and to the pick-upoptics assembly, the main processor being configured with instructionsstored in non-volatile memory to: operate the pick-up optics assembly tosuccessively step through the plurality of spectral filters and obtainat least one two-dimensional image at each spectral filter step, and toactivate and deactivate the illuminators. The image data processor isconfigured with instructions stored in non-volatile memory to: receiveimages corresponding to each spectral filter step, assemble the receivedimages into a hyperspectral image cube, and to process the image cube todetermine at least one color of the sample surface. The hyperspectralimage cube may comprise three dimensions; two dimensions correspondingto the two dimensions of the acquired images and a third dimensioncorresponding to the nominal peak wavelengths of the plurality ofselectable filters. The main processor and the image data processor maybe combined or separate.

The image data processor may be further configured to process the imagecube to determine at least one color of the sample surface by: identifya region of interest; analyze the region of interest to determinewhether a sufficient area of uniform color exists to permit a simplecolor averaging function; if a sufficient area of uniform color exists,perform a simple color averaging process to determine a color of theregion of interest; and if a sufficient area of uniform color does notexist, segment pixels of an image to exclude pixels including shadowedsurfaces and to exclude pixels including specular reflections anddetermine a color of the remaining pixels. The image data processor isfurther configured to process the image cube to determine at least onecolor of the sample surface by segmenting pixels of an image to excludepixels including shadowed surfaces and to exclude pixels includingspecular reflections and determine a color of the remaining pixels for aplurality of hues. A plurality of dominant hues may be identified andpresented to a user. The segmenting may comprise segmenting pixelshaving a lightness lower than the 50^(th) percentile into a first bin,segmenting pixels having a lightness at or above the 95^(th) percentileinto a second bin, and the remaining pixels into a third bin, wherein acolor of the third bin is determined. A plurality of colors may bedetermined for a sample surface by constructing a reflectance curve fora plurality of pixel locations in an acquired image with respect to thenominal peak wavelengths of the plurality of spectral filters, andtransforming each of the reflectance curves into coordinates in athree-dimensional color space.

The illumination assembly may include a plurality of illuminatorsdirected toward the sample measurement area. The plurality of spectralfilters may comprise narrow bandpass filters.

The hyperspectral imaging system may further comprise a glossmeasurement assembly with pick-up optics being directed toward thesample stage and in the same plane as the at least one illuminator.

The image cube may comprise a plurality of two dimensional images, withone axis of the cube corresponding to the wavelengths of the pluralityof spectral filters.

The main processor may be further configured with instructions stored innon-volatile memory to obtain an image under ambient lightingillumination and active illumination at each spectral filter step.

A method for measuring one or more colors of a sample surface, maycomprise: placing a sample to be measured within a measurement area ofan imaging spectrophotometer including a plurality of selectablespectral filters having different nominal peak wavelengths and animaging camera; successively stepping through the plurality of spectralfilters of the pick-up optics assembly and obtaining at least onetwo-dimensional image at each spectral filter step; assembling theimages into a hyperspectral image cube; and processing the image cube todetermine at least one color of the sample surface. The step ofprocessing the image cube to determine at least one color of the samplesurface may further comprise: identifying a region of interest;analyzing the region of interest to determine whether a sufficient areaof uniform color exists to permit a simple color averaging function; ifa sufficient area of uniform color exists, performing a simple coloraveraging process to determine a color of the region of interest; if asufficient area of uniform color does not exist, segmenting pixels of animage to exclude pixels including shadowed surfaces and to excludepixels including specular reflections; and determining a color of theremaining pixels.

The step of processing the image cube to determine at least one color ofthe sample surface further may further comprise segmenting pixels of animage to exclude pixels including shadowed surfaces and to excludepixels including specular reflections and determining a color of theremaining pixels for a plurality of hues. For example, the segmentingmay comprise segmenting pixels having a lightness lower than the 50^(th)percentile into a first bin, segmenting pixels having a lightness at orabove the 95^(th) percentile into a second bin, and the remaining pixelsinto a third bin, wherein a color of the third bin is determined.

BRIEF DESCRIPTION OF THE DRAWINGS

The following drawings illustrate the concepts of the present invention.Illustrations of an exemplary device are not necessarily drawn to scale.

FIG. 1 is a perspective view of an embodiment of a hyperspectral imagingspectrophotometer and system according to one aspect of the presentinvention.

FIG. 2 is a perspective view of the example of FIG. 1 with the outercasing removed.

FIG. 3 is an exploded view of a pick-up optics assembly of the exampleof FIG. 1 according to another aspect of the present invention.

FIG. 4a is an exploded view of the filter wheel assembly of the exampleof FIG. 3.

FIG. 4b is a side view of the filter wheel assembly of the example ofFIG. 3.

FIG. 5a is an exploded view of a gloss measurement assembly of theexample of FIG. 1.

FIG. 5b is cross section view of a gloss measurement assembly of theexample of FIG. 1.

FIG. 6 is a block diagram of a hyperspectral imaging spectrophotometersystem of the present invention in combination with a personal computeraccording to another aspect of the present invention.

FIG. 7 is a view of the calibration slider of the example of FIGS. 1 and2.

FIG. 8 is a flow chart of a process of building a hyperspectral imagecube according to another aspect of the present invention.

FIG. 9 is a flow chart of a process of outputting spectral data of asample according to another aspect of the present invention.

FIG. 10 is a flow chart of a monochrome color measurement process of aspectral image according to another aspect of the present invention.

DETAILED DESCRIPTION

While the embodiments described can take many different forms, specificembodiments illustrated in the drawings will be described with theunderstanding that the present disclosure is to be considered anexemplification of the principles of the invention, and is not intendedto limit the invention to a specific embodiment illustrated.

A hyperspectral imaging spectrophotometer system is described. Ratherthan reduce a given sample to a single color measurement, thehyperspectral imaging spectrophotometer system acquires successive twodimensional images of a sample through a series of narrow band spectralfilters, and then processes the images to provide a three-dimensionalhyperspectral reflectance data cube (two spatial dimensions plus aspectral reflectance data dimension).

The hyperspectral imaging spectrophotometer system comprises an imagingspectrophotometer and a computer for image processing. In a preferredembodiment, the imaging spectrophotometer and the computer are embodiedin the same instrument. The computer for image processing may comprise anon-volatile memory device comprising computer-readable instructions forimage processing, for example as an image processing module. Thenon-volatile memory device may comprise an operating system, for examplea Linux operating system, for example a real-time operating system.

The imaging spectrophotometer acquires two-dimensional colorimetricmeasurements to form image data of a sample, for example using animaging sensor, and the computer uses transforms to convert the imagedata into a colorimetric representation that closely matches the humanvisual perception of physical samples, for example physical samples thatmay comprise visual textures or patterns. The imaging spectrophotometeris calibrated for color measurements and collects a plurality oftwo-dimensional (spatial dimensions) measurement frames of the sample ata plurality of wavelengths of light, respectively, to create athree-dimensional true spectral color image that captures spatialcolorimetric features. The third-dimension is color data (wavelength)associated with each spatial location. The three-dimensionalmeasurements, or image cube, is then analyzed for sample uniformity,texture, statistical artifacts, color groupings, and spatial complexityin order to produce transformed color information that more closelymatches human visual perception.

With the described system, there is no need for any part of thehyperspectral imaging spectrophotometer system to contact a sample whilemaking measurements, and the non-contact ability helps match paintsamples that are not completely dry. In some embodiments, the system isdesigned to reject the contribution of ambient light illumination to themeasurements. In some embodiments, the system is designed to exchangedata with a tethered host computer running an application, for examplean application that allows the user to preview and select the sampleregion being measured.

FIG. 1 is a perspective view of an embodiment of a hyperspectral imagingspectrophotometer 10, including an outer casing 12 and a calibrationslider assembly 14. FIG. 2 is a perspective view of the hyperspectralimaging spectrophotometer 10 with the outer casing 12 removed. FIG. 2shows the calibration slider assembly 14, an illumination assembly 16, agloss measurement assembly 18, a pick-up optics assembly 20, a mainprocessor 22, and an image data processor 24. These assemblies andcircuit boards may be mounted directly or indirectly on a chassis 26.The illumination assembly 16 may comprise a plurality of illuminators28. The main processor 22 and the image data processor may be separateor combined in a single processor circuit board.

FIG. 3 is an exploded view of the pick-up optics assembly 20, comprisinga lower lens assembly 30 and a spectral imaging assembly 40. The lowerlens assembly 30 may comprise a lower lens mount 32, a window retainer33 which holds an Infra-Red/Near Infra-Red block window 34, a lensretainer 36 which holds a lens 38, and screws 39. The lens 38 maycomprise a front doublet lens. A spectral imaging assembly 40 maycomprise a filter drive assembly 42, an upper lens assembly 44, opticscover 46 and a camera printed circuit board (PCB) 48. The upper lensassembly 44 may comprise a rear lens thread 52 holding a focus lens 54and a focus lens mount 56. The focus lens 54 may comprise a rear doubletlens. The upper lens assembly 44 may be held in place in the opticscover 46 by a set screw 58. The filter drive assembly 42 may be mountedon a motor mount 62. A dust shield 64 may be disposed between a filterwheel 72 and the motor mount 62. The filter drive assembly 42 may alsoinclude a side sensor PCB assembly 66 and flexible cable 68.

The camera PCB 48 includes an image sensor 49 and related electronics,including firmware and/or software, for processing and transferringimages to a computer. The image sensor 49 may be a CMOS sensor, CCDsensor, or other two-dimensional image sensor. The image sensor may bemonochromatic or multispectral, such as RGB. In one example, imagesensor 49 comprises a 4-8 megapixel RGB CMOS sensor.

FIG. 4a is an exploded view of the filter drive assembly 42, and FIG. 4bis a side view of the filter drive assembly 42. The filter driveassembly 42 comprises a filter wheel 72, a filter wheel actuator 74, andfilter wheel controller 76. In one example, the filter wheel has 32positions with 31 apertures. In some embodiments, thirty of theapertures include color filters. In some embodiments, the remainingaperture may have no filter and may be used as an open air measurementposition (i.e., non-filtered measurement). The position without anaperture is a closed position, completely blocking light, to allow formeasurement of the image sensor's dark current.

The filters exist to provide a reflectance measurement at specificwavelengths. For example, the filter wavelengths may be evenly spacedbetween 400 nm and 700 nm. In this example, the filters may havepassband functions with nominal peaks every 10 nm. Fewer filters may beused for lower resolution spectral measurement, and more filters may beused for higher resolution spectral measurements. Fewer filters mayallow for fewer images to be taken and to make a complete set ofmeasurements faster, at a cost of some precision. Fewer filters may besuitable such as in connection with ambient light correction relatedmeasurements, especially if ambient light levels are weak. Interpolationof measurements acquired at a plurality of filter wavelengths may beused to estimate reflectance at wavelengths between filter measurementbands. Some embodiments of the hyperspectral imaging spectrophotometersystem may comprise a filter wheel, for example because a filter wheelmay have a small form factor, but that is only one example of how topresent filters individually. The filters may be provided in alternativestructures, such as a linear array of filters, for example.

The filter wheel actuator 74 may comprise a stepper motor. The filterwheel controller 76 may comprise circuitry including a stepper motorcontroller IC and motor home sensor. The illumination assembly 16 andpick-up optics assembly 20 are arranged such that the lower lensassembly 30 receives light reflected from a measurement surface andfocuses the light on an aperture (or the closed position) of the filterwheel 72. The upper lens assembly 44 focuses light passing through anaperture on the filter wheel 72 (filtered or unfiltered) onto the imagesensor 49. The optics may provide any suitable field of view andmagnification. For example, the field of view may be in the range of15-18 mm, and the magnification may be in the range of 0.224 to 0.274.

While a 4-8 megapixel RGB image sensor facilitates obtaining a highresolution live view for sample positioning, the high resolution mayresult in an increased processing burden for the resultant image cube.For example, one example of an RGB sensor has full raw resolution of2608×1952 pixels. Coupled with the optics above, this would result in afull sensor resolution of 96 pixels/mm. This resolution is higher thannecessary to ascertain colors for most samples being measured. Theresolution (and processing burden) may be reduced for each imagecorresponding to a spectral measurement with “binning” and “skipping.”For example, the image sensor 49 may be configured to average together4×4 red pixels, 4×4 green pixels, and 4×4 blue pixels into groupedpixels at an image resolution of 652×488 pixels (binning), therebyreducing resolution to 24 pixels/mm. Also, pixels not responsive to agiven selected narrow band filter used to acquire a given image may beskipped. The grouped pixels may then be further reduced to one grayscale value. This results in a gray measurement resolution of 12pixels/mm.

FIG. 5a is an exploded view of the gloss measurement assembly 18 andFIG. 5b is a cross section view of the gloss measurement assembly 18.The gloss measurement assembly 18 may comprise gloss barrel 80, glosspick-up lens 82, gloss lens retainer 84, gloss filter 86, IR filterglass 88, lens retainer 90 wave ring 92, and gloss PC board assembly 94.The gloss PC board assembly 94 includes a gloss sensor, such as a photodiode or a photodiode array. The gloss measurement assembly 18 may belocated to be collinear with one of the illuminators 28. For example,where an angle of illumination to a surface to be measured defines aplane of incidence, the gloss measurement assembly 18 may be in theplane of incidence of one of the illuminators 28. Gloss measurements maybe made by the gloss measurement assembly 18 in combination with theilluminator 28 opposite of it and in the same plane.

The illumination assembly 16 may comprise a plurality of illuminators28. Each illuminator 28 may comprise one or more of LED, incandescent,fluorescent, arc, flash or other suitable light sources. For example, anilluminator 28 may comprise a broadband white LED or, preferably,independently controllable red, green and blue LEDs. Preferably, the LEDilluminators may be regulated for one or more of temperature andbrightness . In the illustrated example seen in FIG. 2, there are threeLED illuminators 28 spaced 120° apart in azimuth. The LED illuminators28 are also angled 45° in elevation with respect to a subjectmeasurement area, such as sample window 15 or calibration target surface19. This provides a 45/0 measurement geometry. Other illumination anglesand measurement geometries may also be used, including sphericallydiffuse illumination. The illuminators may be designed to providesearchlight, or collimated, illumination in order to minimize depth offield errors at the sample plane. The LEDs in the illuminators 28 may beoperated in continuous illumination mode or the LEDs may be modulated,such as with pulse width modulation.

Referring to FIG. 7, the calibration slider assembly 14 includes asample window 15 a calibration target support 17 (phantom lines) and acalibration target surface 19. The calibration target support includes aslider mechanism which includes a measurement position, a calibrationposition and a recessed position (for cleaning). When the sample window15 or calibration target surface 19 are positioned under the pick-upoptics assembly 20, they are within the measurement area of thehyperspectral imaging spectrophotometer 10. The measurement positionassists with correct targeting in a desired location on a sample. Thecalibration target position also provides surfaces of known reflectance,such as calibration target surface 19, which may be used via correlationto characterize the illumination being applied to the sample beingmeasured. The calibration slider assembly 14 may also comprise one ormore base surfaces, pads, or rails 15 a near the underside of the samplewindow 15 a to flatten paper samples, textile samples, and the like. Inaddition to sliding mechanisms, additional structures may be used tomove a calibration target into view and out of view of the measurementoptics.

The calibration target surface may comprise a conventional targetsurface, such as made of porcelain on steel materials. Some embodimentsmay comprise a push-broom scanner, where sensors measure a line ofpixels, and then translation of either the scanner or the sample allowssuccessive measurements to build up data in two spatial dimensions.

In addition to the assemblies described above, additional assemblies maybe included, such as a power assembly, a user interface 95 comprising,for example, a button (switch) assembly for initiating a measurement andstatus LEDs. The hyperspectral imaging spectrophotometer system 10 mayalso include one or more communication ports 96 for connecting thehyperspectral imaging spectrophotometer 10 to a personal computer (PC)98. The communication ports may include wired (e.g., USB) or wirelesstechnologies (e.g., Bluetooth, WiFi). Additional accessories may beprovided externally from the outer casing 12, such as gray backing onwhich a sample may be placed for translucence measurement and additionalcolor calibration tools (e.g., color checker tiles and references).

The main processor 22 is coupled to and configured with appropriatefirmware instructions to control and operate the illumination assembly16, gloss measurement assembly 18, and pick-up optics assembly 20according to the methods described herein. The main processor 22 furthercommunicates acquired spectral image data to the image data processor24.

In an example illustrated in FIG. 6, a personal computer (PC) 98 isprovided to be used in connection with the imaging spectrophotometer.The PC 98 includes software to provide the following functionality:provide a user with the ability to select a region of interest (ROI),and return measureable attributes of the ROI. The measurable attributesmay include color, texture, translucence and shape, for example one ormore of shape dimensions and area. The PC 98 may also provide tiltcorrection to sample images. The PC 98 may also provide relatedfunctionality, such as electronic color standard fan deck lookup in alocally or remotely stored database to allow users to select a colorfrom a pre-existing color library.

The present invention may be used in a workflow where a user would liketo measure the color of a physical object, and then render or reproducethat color as in a mixture of paint, ink, or other colorant set suchthat the reproduced coloring agent when applied to various objectsprovides an acceptable visual match to the original physical object. Theuser's physical object color may be highly uniform, or may containspatial color patterns or patches, for example contiguous patches, wherethe color of interest is represented.

The multispectral imaging spectrophotometer hardware and processingalgorithms provide accurate color match on many textured and non-planarsamples by controlling the effect of specular glints, shadowing, andother artifacts, effectively removing them from the measurements.Hardware and algorithms (as described in more detail below) also allowextraction of desired color from multi-colored samples.

Referring to FIG. 8, in one example of a hyperspectral measurementprocess 100, the main processor 22 provides signals to the filter wheelcontroller 76 to step the filter wheel 72 through each filter wheelposition while also controlling the illumination assembly 16 and thecamera PCB 48 to acquire one or more images at each wheel position. Forexample, a user may initiate a measurement in step 102. The mainprocessor 22 may optionally advance the filter wheel 72 to the open (nofilter position) and operate the camera PCB 48 in a live view video modein step 104. The main processor 22 may communicate the live view videoof the image sensor's field of view to a connected personal computer(PC) 98. This facilitates placement and orientation of the samplesurface to be measured. The main processor 22 may optionally advance thefilter wheel 72 to the closed position and deactivate the illuminationassembly 16, and an image may be acquired to measure the image sensor'sdark current. The main processor 22 may then advance the filter wheel 72to a first filter position in step 106. The main processor 22 may causethe camera PCB 48 to acquire an image with illuminators 28 off foracquisition of an ambient lighting image in step 108, and another imagewith illuminators 28 on to obtain an active illumination image in step110. Comparison of ambient illumination images with illuminated imagesallows for correction for ambient lighting. In step 112, main processor22 determines whether images have been taken at all filter positions. Ifnot, the filter wheel 72 is advanced to the next position in step 104and the measurements repeated. The filter wheel 72 may be successivelyadvanced through each of the filter window positions and the openposition. In this way, the hyperspectral imaging spectrophotometer 10acquires thirty different wavelengths of spectral reflectance data,along with surface texture reflectance data at each wavelength. When allof the different measurements are obtained, the image cube is assembledin step 114 by image data processor 24. The image cube may comprise aset of grey scale images, where two dimensions (x, y) of the cubecorrespond to the field of view of the image sensor 49 (e.g., an imageof the surface being measured), and a third dimension (z) corresponds toa plurality of measurements at different wavelengths. However, the imagecube need not be represented as an actual or virtual cube. The imagecube may comprise a three-dimensional array of pixel values, where twodimensions represent an array of pixels corresponding to an image of asample being measured, and a third dimension of the array corresponds toa given pixel over the range of measurement wavelengths. Additionaldimensions to the array may be added for additional measured reflectanceproperties.

At the open, non-filtered position the illuminators may be activated fora gloss measurement by the camera PCB 48. The illuminators 28 may alsobe activated individually and separate images acquired from eachillumination direction for additional gloss/texture measurements. Theimages acquired by the camera PCB 48 may be combined with glossmeasurements made by the gloss measurement assembly 18.

A method for determining the colors to report from an image cube asmeasured and assembled according to the example of FIG. 8 may bestructured around two major steps: identification of the significantcolors in the image, and calculation of the values in a standard colorspace (e.g., CIELAB, CIEXYZ, reflectance, SRGB), that optimallycorrespond to the visual perception of the color, for each significantcolor.

There are multiple reasons for the presence of multiple colors in theimage of an object, and there are multiple methods for calculation ofthe optimal color values for a color. An object such as wood flooringmay naturally contain multiple colors. An object may also be produced tohave multiple colors, for example, textiles made from multiple colorthreads, or printed images. And the physical texture of an object maycause an image of it to have different colors, due to variations inshadows and specular reflections.

The identification of the significant colors in an image starts withrepresenting the 3D image cube as color values in a standard colorspace, and then determining the histogram of these color values. Forexample, a reflectance curve may be constructed for each pixel (such asthe grouped pixels) corresponding to a measured spot on a sample surfaceby applying intensity values corresponding to each of the measurementsat each spectral filter wavelength. These reflectance curves may betransformed into color coordinates in a three-dimensional color spacethrough conversion processes known to persons of skill in the art. Thethree dimensional color space may comprise, for example, the CIELAB orCIEXYZ color spaces. In the example provided above, this would result in12 individual color measurements per square millimeter along a directionon the surface being measured. A color histogram may be determined fromthese values.

To extract colors from a complex measured sample, a starting point is toidentify local maxima in the histogram. Some images with multiple colorsmay not have color distributions with well-defined peaks for each color.For these images, a thresholding method, such as Otsu's method or anentropy based method, can separate the histogram into multiple regionsof color space, even without a local maximum inside the region.

The calculation of the optimal color in an image of an object to matchthe visual perception of the object has multiple steps. The basic stepis to identify the region of interest in the image, whose color bestmatches the visual perception of the object. This region of interest isnot necessarily contiguous, and images in which the region of interestis not contiguous are the images for which the hyperspectral imagingspectrophotometer 10 may be the most valuable. That is because anoptimal color may be accurately measured by performing a traditionalspectrophotometer spot measurement (which averages color over the entiremeasurement area of the spectrophotometer) if the object has asufficiently large region of uniform color.

The calculation of the region of interest will be performed in colorspace, starting with the data previously used to identify the presenceof the color. Studies of the human visual process have found that whenit recognizes objects in a scene, it will perceive the color as a singleproperty of the object, even though it may be sensing multiple colorsfrom different regions of the object, due to shadows, specularhighlights, or other effects. Accordingly, a region in color space thatoptimally matches a perceived object color may be calculated.

For the above approach to be successful, the color which matches humanperception of the object must be present in the images. Some additionalsteps may be included in the method when this condition is not met.Steps that modify the image to more closely match perceived color areused in combination with the basic region of interest step describedabove.

One such optional step is to calculate a lightness correction factor tobe applied to an image, for example when every pixel in the imageincludes some shadowing. This may be the situation for samples that haveparticularly high levels of physical texture. The specific value of thescale factor will also depend on illumination supplied to the sample bythe hyperspectral imaging spectrophotometer 10. For example, a largerscale factor is likely to be needed when using a 45° illuminationgeometry than when using diffuse illumination geometry. Such a scalefactor may be determined based on the overall variation in lightness inthe color region under consideration.

Another optional step to modify color values from the image is one thatcompensates for multiple opposing colors in the image. The perceptualeffect that such methods correspond to is the effect of surroundingcolors on perception of the color at a location in an image. Models thatcorrect for such perceptual effects are known as image color appearancemodels. While the initial motivation for such models was to produce anatural appearance in the display of photographic images, in this methodthey are used to determine the perceived color based on a measuredimage.

A step related to the method as described above is a step to evaluatethe uniformity of the image, to determine whether to use a simpleaverage of the measured values, or to use the method described above.Ideally, even if it is reasonable to use a simple average of measuredvalues, the method described above should provide values that are withinthe usual color tolerances for such measurements. The advantage todetecting whether an image is sufficiently uniform to use simpleaveraging is to avoid the increased processing time with the colorextraction method.

In some embodiments, the system may operate as a stand-alone instrumentthat provides a user interface. For example, the system may comprise oneor more built-in displays and buttons to allow user interactionincluding selecting a region of interest, a built-in touch-screendisplay to allow user interaction including selecting a region ofinterest, or wireless communication to a computer or a mobile device toallow user interaction including selecting a region of interest. Oneexample of allowing a user to select a region of interest can be asimple user-selectable measurement area, such as a circular shape of anysize defined by the user. As another example, the measurement areaselectable by the user may comprise any arbitrary set of pixels defined,for example designated via a touch-sensitive display, by the user.

The hyperspectral imaging spectrophotometer 10 preferably performs imageprocessing on the acquired images as discussed above, for example, bythe image data processor 24. The image data processor 24 may comprise acomputer module having a processor and memory, including an operatingsystem, for example a Linux operating system, that provides thecomputing and image processing onboard. Instructions stored in firmwareon the image data processor 24 provide image processing capabilitiesincluding ambient light rejection, calibration algorithms calculatingreflectance from raw image cubes, color extraction from complex samplesthrough processes described herein, including automatic detection ofmultiple colors in a region of interest, automatic selection of multipleregions of the same color, sorting of colors in the region of interestto obtain reflectance measurements for each color, for example eachcolor filter, and production of a histogram of the region of interest.Ambient light rejection may be provided by taking measurements with andwithout illumination from the illumination assembly and then performingsubtraction (such as digitally) to eliminate ambient light. Physicalshielding may also be used to prevent ambient light from illuminatingthe sample being measured.

As illustrated in the FIG. 9 flow chart, an example of a process 120 forextracting one or more colors from an inspiration sample may proceed asfollows: A user may initiate a hyper spectral measurement of a sample instep 122 as described with respect to FIG. 8 above. A Region of Interest(ROI) is identified in step 124. An initial analysis is performed todetermine XYZ/LCH variance of the ROI in step 126. If the ROI has asufficiently large region of uniform color, a decision may be made toperform a simplified color analysis in step 128. If not, then adetermination is made to apply a monochrome process (FIG. 10) or amulticolor process in step 130. When the sample has a color that isrelatively uniform, the monochrome process may be selected in step 132to reduce processing time. When the sample has two or more hues, amulticolor process may be applied in step 134. The multicolor processmay involve applying the monochrome process to more than one hue. Wheneach hue has been processed, a palette of multiple colors (for example,two to four dominant colors) may be presented to the user in step 136.Colors may be ranked from most dominant to least dominant. The user mayselect a desired color from the palette in step 138. At the end of eachprocess, spectral data for the measured color may be output in step 140.

Referring to FIG. 10, the monochrome process 150 may involve thefollowing steps. The image may be segmented using percentiles oflightness or luminance in step 152. The minimum lightness may be the50^(th) percentile or greater. The maximum lightness may be set by oneof the following: chrome versus lightness peak, 95^(th) percentilelightness, excluding highest lightness bin, or by difference overmedian. Pixels having a measured intensity that is above a certainthreshold, for example above 95 percent of intensity range, mayrepresent specular reflections and may be excluded from colormeasurement in step 154, and pixels that are lower than the 50^(th)percentile may represent shadows from surface texture and may beexcluded from color measurement in step 156.

The remaining pixels are analyzed to determine a color value for theimage in step 158. Pixels that are not excluded from color measurementmay be segmented or clustered into bins. For example, pixels may beclustered based on hue, chroma and lightness. The bins may then beranked by the number of pixels in each bin. Bins with the highest numberof clustered pixels may be considered the dominant colors of themeasurement. An example of one process for determining color once pixelsare segmented into bins and ranked for dominance is a simple average.Another example is a weighted average, based on location in color space.For a basic monochrome process, this is an applied direction to themeasured values from the 3D image cube. If lightness correction isnecessary, the lightness correction factor may be used to scale thereflectance values at this point in the process or before applying themonochrome process.

As a first example of one approach for an appearance/opposing colorscompensation process, it is noted that the appearance compensationprocess produces a transformation of the image in XYZ space. Adifference in average XYZ values is determined between the untransformedand the appearance-transformed color values in the XYZ color space. Thena corresponding difference in spectral reflectance values is calculatedto match the difference in XYZ values, with the calculated spectraldifference as a linear combination of spectral reflectance values ofpixels in the image. At this point, the process continues just as withthe monochrome process, but using the transformed 3D image cube ratherthan the raw 3D image cube.

A second example of an approach for an appearance/opposing colorscompensation process would be to calculate a spectral difference basedon the spectral curves of the colorants in a formulation system. A thirdexample would be to match the untransformed reflectance values, but touse the difference between untransformed and transformed color values inthe target metric for formulation. The second and third examples can beimplemented when the described instrument is part of a formulationsystem, but the first example is applicable to a stand-alone instrument.

The described system has uses in addition to retail paint matching,especially in industrial labs where color is measured on textured orshaped samples. In some embodiments, the system is tethered to acomputer, does not have an on-screen display, and uses a softwareapplication for color selection from an image, automatic calibration,on-screen targeting, line-of-sight targeting, or bar code scanning, forexample.

From the foregoing, it will be understood that numerous modificationsand variations can be effectuated without departing from the true spiritand scope of the novel concepts of the present invention. It is to beunderstood that no limitation with respect to the specific embodimentsillustrated and described is intended or should be inferred.

1. A hyperspectral imaging system for measuring one or more colors of asample surface, comprising: a sample measurement area; an illuminationassembly including at least one illuminator directed toward the samplemeasurement area; a pick-up optics assembly including a plurality ofselectable spectral filters having different nominal peak wavelengthsand an imaging camera, where a field of view of the imaging cameraincludes at least a portion the sample measurement area; a mainprocessor coupled to the illumination assembly and to the pick-up opticsassembly, the main processor being configured with instructions storedin non-volatile memory to: operate the pick-up optics assembly tosuccessively step through the plurality of spectral filters and obtainat least one two dimensional image at each spectral filter step, and toactivate and deactivate the illuminators; and an image data processorbeing configured with instructions stored in non-volatile memory to:receive images corresponding to each spectral filter step, assemble thereceived images into a hyperspectral image cube, and to process theimage cube to determine at least one color of the sample surface, thehyperspectral image cube having two dimensions corresponding to the twodimensions of the acquired images and a third dimension corresponding tothe nominal peak wavelengths of the plurality of selectable filters. 2.The hyperspectral imaging system of claim 1, wherein the image dataprocessor is further configured to process the image cube to determineat least one color of the sample surface by: identify a region ofinterest; analyze the region of interest to determine whether asufficient area of uniform color exists to permit a simple coloraveraging function; if a sufficient area of uniform color exists,perform a simple color averaging process to determine a color of theregion of interest; if a sufficient area of uniform color does notexist, segment pixels of an image to exclude pixels including shadowedsurfaces and to exclude pixels including specular reflections anddetermine a color of the remaining pixels.
 3. The hyperspectral imagingsystem of claim 2, wherein the image data processor is furtherconfigured to process the image cube to determine at least one color ofthe sample surface by segmenting pixels of an image to exclude pixelsincluding shadowed surfaces and to exclude pixels including specularreflections and determine a color of the remaining pixels for aplurality of hues.
 4. The hyperspectral imaging system of claim 3,wherein a plurality of dominant hues are identified and presented to auser.
 5. The hyperspectral imaging system of claim 3, wherein thesegmenting comprises segmenting pixels having a lightness lower than the50^(th) percentile into a first bin, segmenting pixels having alightness at or above the 95^(th) percentile into a second bin, and theremaining pixels into a third bin, wherein a color of the third bin isdetermined.
 6. The hyperspectral imaging system of claim 1, wherein theillumination assembly includes a plurality of illuminators directedtoward the sample measurement area
 7. The hyperspectral imaging systemof claim 1, further comprising a gloss measurement assembly with pick-upoptics being directed toward the sample stage and in the same plane asthe at least one illuminator.
 8. The hyperspectral imaging system ofclaim 1, wherein the plurality of spectral filters comprise narrowbandpass filters.
 9. The hyperspectral imaging system of claim 1,wherein the plurality of spectral filters comprise narrow bandpassfilters with nominal centers spaced 10 nm apart.
 10. The hyperspectralimaging system of claim 1, wherein a plurality of colors are determinedfor a sample surface by constructing a reflectance curve for a pluralityof pixel locations in an acquired image with respect to the nominal peakwavelengths of the plurality of spectral filters, and transforming eachof the reflectance curves into coordinates in a three-dimensional colorspace.
 11. The hyperspectral imaging system of claim 1, wherein the mainprocessor is further configured with instructions stored in non-volatilememory to obtain an image under ambient lighting illumination and activeillumination at each spectral filter step.
 12. A method for measuringone or more colors of a sample surface, comprising: placing a sample tobe measured within a measurement area of an imaging spectrophotometerincluding a plurality of selectable spectral filters having differentnominal peak wavelengths and an imaging camera; successively steppingthrough the plurality of spectral filters of the pick-up optics assemblyand obtaining at least one two dimensional image at each spectral filterstep; assembling the images into a hyperspectral image cube, thehyperspectral image cube having two dimensions corresponding to the twodimensions of the acquired images and a third dimension corresponding tothe nominal peak wavelengths of the plurality of selectable filters; andprocessing the image cube to determine at least one color of the samplesurface.
 13. The method of claim 12, wherein the step of processing theimage cube to determine at least one color of the sample surface furthercomprises: identifying a region of interest; analyzing the region ofinterest to determine whether a sufficient area of uniform color existsto permit a simple color averaging function; if a sufficient area ofuniform color exists, performing a simple color averaging process todetermine a color of the region of interest; if a sufficient area ofuniform color does not exist, segmenting pixels of an image to excludepixels including shadowed surfaces and to exclude pixels includingspecular reflections; and determining a color of the remaining pixels.14. The method of claim 13, wherein the step of processing the imagecube to determine at least one color of the sample surface furthercomprises segmenting pixels of an image to exclude pixels includingshadowed surfaces and to exclude pixels including specular reflectionsand determining a color of the remaining pixels for a plurality of hues.15. The method of claim 14, wherein the segmenting comprises segmentingpixels having a lightness lower than the 50^(th) percentile into a firstbin, segmenting pixels having a lightness at or above the 95^(th)percentile into a second bin, and the remaining pixels into a third bin,wherein a color of the third bin is determined.
 16. The method of claim14, further comprising identifying a plurality of dominant hues andpresenting the dominant hues to a user.
 17. The method of claim 12,wherein the plurality of spectral filters comprise narrow bandpassfilters.
 18. The method of claim 12, wherein the image cube comprises aplurality of two-dimensional images, with one axis of the cubecorresponding to the wavelengths of the plurality of spectral filters.19. The method of claim 12, wherein the step of obtaining at least oneimage at each spectral filter step further comprises obtaining an imageunder ambient lighting illumination and active illumination at eachspectral filter step.
 20. The method of claim 12, wherein a plurality ofcolors are determined for a sample surface by constructing a reflectancecurve for a plurality of pixel locations in an acquired image withrespect to the nominal peak wavelengths of the plurality of spectralfilters, and transforming each of the reflectance curves intocoordinates in a three dimensional color space.